Method and apparatus for switching between multiple programs by interacting with a hyperlinked television broadcast

ABSTRACT

A system and method of adding hyperlinked information to a television broadcast. The broadcast material is analyzed and one or more regions within a frame are identified. Additional information can be associated with a region, and can be transmitted in encoded form, using timing information to identify the frame with which the information is associated. The system comprising a video source and an encoder that produces a transport stream in communication with the video source, an annotation source, a data packet stream generator that produces encoded annotation data packets in communication with the annotation source and the encoder, and a multiplexer system in communication with the encoder and the data packet stream generator. The encoder provides timestamp information to the data packet stream generator and the data packet stream generator synchronizes annotation data from the annotation source with a video signal from the video source in response to the timestamp information. The multiplexer generates a digital broadcast signal that includes an augmented transport stream from the transport stream and the encoded data packets. A receiver displays the annotation information associated with the video signal in response to a viewer request on a frame by frame basis. A viewer can respond interactively to the material, including performing commercial transactions, by using a backchannel that is provided for interactive communication.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. provisional patentapplications serial No. 60/185,668, filed Feb. 29, 2000, entitled“Interactive Hyperlinked Video System”; Ser. No. 60/229,241, filed Aug.30, 2000, entitled “A Method and Apparatus for Hyperlinking in aTelevision Broadcast”; and Ser. No. 60/233,340, filed Sep. 18, 2000,entitled “A Method and Apparatus for Hyperlinking in a TelevisionBroadcast.” The entirety of each of said provisional patent applicationsis incorporated herein by reference.

FIELD OF THE INVENTION

[0002] The invention relates to the field of broadcast television andmore specifically to the field of hyperlinking in a televisionbroadcast.

BACKGROUND OF THE INVENTION

[0003] Broadcasts of information via television signals are well knownin the prior art. Television broadcasts are unidirectional, and do notafford a viewer an opportunity to interact with the material thatappears on a television display. Viewer response to material displayedusing a remote control is known but is generally limited to selecting aprogram for viewing from a listing of available broadcasts. Inparticular, it has proven difficult to create hyperlinked televisionprograms in which information is associated with one or more regions ofa screen. The present invention addresses this need.

SUMMARY OF THE INVENTION

[0004] The invention provides methods and systems for augmentingtelevision broadcast material with information that is presented to aviewer in an interactive manner.

[0005] In one aspect, the invention features a hyperlinked broadcastsystem. The hyperlinked broadcast system includes a video source and adata packet stream generator that produces a transport stream incommunication with the video source. The system includes an annotationsource, a data packet stream generator that produces encoded annotationdata packets in communication with the annotation source and thegenerator, and a multiplexer system in communication with the encoderand a data packet stream generator. The multiplexer generates a digitalbroadcast signal that includes an augmented transport stream from thetransport stream from the video source and the encoded data packets. Theencoder provides timing information to the data packet stream generatorand the data packet stream generator synchronizes annotation data fromthe annotation source with a video signal from the video source inresponse to the timing information.

[0006] In one embodiment, the annotation information includes mask dataand at least one of textual data and graphics data. In one embodiment,the mask data includes location and shape information of an object in anannotated video frame.

[0007] In another aspect, the invention features a hyperlinked broadcastand reception system. The hyperlinked broadcast and reception systemincludes a video source, an encoder that produces a transport stream incommunication with the video source, an annotation source, and a datapacket stream generator that produces encoded annotation data packets incommunication with the annotation source and the generator. The systemalso includes a multiplexer system in communication with the encoder andthe data packet stream generator. The multiplexer generates a digitalbroadcast signal comprising an augmented transport stream from thetransport stream and the encoded data packets. The system additionallyincludes a broadcast channel in communication with the multiplexersystem, a receiver in communication with the broadcast channel, and adisplay device in communication with the receiver. The encoder providestiming information to the data packet stream generator and the datapacket stream generator synchronizes annotation data from the annotationsource with a video signal from the video source in response to thetiming information. The receiver displays the annotation informationassociated with the video signal in response to a viewer request on aframe by frame basis.

[0008] In still another aspect, the invention features a hyperlinkedreception system that includes a receiver in communication with abroadcast channel, and a display device in communication with thereceiver, wherein said receiver displays said annotation informationassociated with a video signal, in response to a user request, on aframe by frame basis, said annotation information being associated withsaid video signal in response to timing information.

[0009] The foregoing and other objects, aspects, features, andadvantages of the invention will become more apparent from the followingdescription and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0010] FIGS. 1A-1D depict a series of frames of video as produced by thesystem of the invention;

[0011]FIG. 2 is a block diagram of an embodiment of a hyperlinked videosystem constructed in accordance with the invention;

[0012]FIG. 2A is a block diagram of the flow of data in the embodimentof the system shown in FIG. 2;

[0013]FIG. 2B is a diagram of a mask packet set;.

[0014]FIG. 2C is a diagram of an initial encoded data packet stream;

[0015]FIG. 2D is a diagram of a final encoded data packet stream;

[0016]FIG. 3 is a block diagram of an embodiment of the multiplexersystem shown in FIG. 1;

[0017]FIG. 4 is a block diagram of an embodiment of the digital receivershown in FIG. 2;

[0018]FIG. 5 is a diagram of an embodiment of the data structures usedby the system of FIG. 2 to store annotation data;

[0019]FIG. 5A is a block diagram of an object properties table datastructure and a program mapping table data structure;

[0020]FIG. 6 is a state diagram of the data flow of an embodiment of thesystem shown in FIG. 2;

[0021]FIG. 7 depicts the interactions between and among states withinthe state machine depicted in FIG. 6 of an embodiment of the invention;

[0022]FIGS. 8A through 8G depict schematically various illustrativeexamples of embodiments of an interactive content icon according to theinvention;

[0023]FIGS. 9A through 9D depict illustrative embodiments of compressionmethods for video images, according to the principles of the invention;

[0024]FIG. 10A shows an exemplary region of a frame and an exemplarymask, that are used to describe a two-dimensional image in the terms ofmathematical morphology, according to the invention;

[0025]FIG. 10B shows an exemplary resultant image of a two-dimensionalmathematical morphology analysis, and a single resultant pixel,according to the principles of the invention; and

[0026]FIG. 11A shows a sequence of exemplary frames and an exemplarymask, that are used to describe a three-dimensional image in the termsof mathematical morphology using time as a dimension, according to theinvention;

[0027]FIG. 11B shows an exemplary resultant frame of a three-dimensionalmathematical morphology analysis using time as a dimension, and a singleresultant pixel, according to the principles of the invention;

[0028]FIG. 11C is a flow diagram showing an illustrative process bywhich three-dimensional floodfill is accomplished, according to oneembodiment of the invention;

[0029]FIG. 12 is a diagram showing an exemplary application ofmathematical morphology analysis that creates an outline of a region,according to the principles of the invention; and

[0030]FIG. 13 is a diagram showing three illustrative examples of theevolutions of histograms over successive frames that are indicative ofmotion, according to the invention.

DESCRIPTION OF THE PREFERRED EMBODIMENT

[0031] In brief overview, the invention provides a way for annotationinformation to be associated with objects displayed in the frames of abroadcast video and displayed upon command of a viewer. For example,referring to FIG. 1, annotation information, in the form of store, priceand availability information may be associated with a specific shirt 2worn by an actor in a television broadcast (FIG. 1A). To achieve this,the shirt 2 is first identified to the system by a designer operating aportion of the system called the authoring system. The designeridentifies 3 the shirt 2 in a given frame, for example by coloring inthe shirt (FIG. 1B), and the system keeps track of the location of theshirt 2 in the preceding and subsequent frames. The designer alsogenerates the text that becomes the annotation data 5 associated withthe shirt 2. Thus in this example the annotation data may include thenames of stores in which the shirt 2 may be purchased, the price of theshirt 2 and the colors available. The system then denotes that the shirt2 has annotation data associated with it, for example by outlining 4 theshirt 2 in a different color within the frame (FIG. 1C).

[0032] When the show is broadcast to a viewer by the transmissionportion of the system, not only is the video broadcast, but also themask which outlines the shirt 2 and the annotation data whichaccompanies the shirt 2. The receiver portion of the system at theviewer's location receives this data and displays the video frames alongwith masks that outline the objects which have associated annotationdata. In this example the shirt 2 in the video flame is outlined. If theviewer of the broadcast video wishes to see the annotation data, he orshe simply uses the control buttons on a standard remote control handsetto notify the receiver portion of the system that the display ofannotation data is desired. The system then displays the annotation data5 on the screen along with the object (FIG. 1D). In this way denotedobjects act as hyperlinks to additional information.

[0033] Referring to FIG. 2, a hyperlinked video broadcast systemconstructed in accordance with the invention includes a transmissionportion 10, a communications channel portion 12 and a reception portion14. The transmission portion 10 includes a video source 20, an authoringtool 24, a database storage system 28 and a transmitter 32. The videosource 20 in various embodiments is a video camera, a video disk, tapeor cassette, a video feed or any other source of video known to oneskilled in the art. The authoring tool 24, which is an annotationsource, receives video data from the video source 20 and displays it fora video designer to view and manipulate as described below. Annotationdata, for example, the text to be displayed with the video image, isstored in an object database 28 and sent to the transmitter 32 fortransmission over the communications channel portion 12 of the system.

[0034] The transmitter 32 includes a video encoder 36, a data packetstream generator 40, and a multiplexer (mux) 44 which combines thesignals from the video encoder 36 and the data packet stream generator40 for transmission over the communications channel 12. The videoencoder 36 may be any encoder, such as an MPEG or MPEG2 encoder, forproducing a transport stream, as is known to one skilled in the art. Thedata packet stream generator 40 encodes additional data, as describedbelow, which is to accompany the video data when it is transmitted tothe viewer. The data packet stream generator 40 generates encoded datapackets. The mux 44 produces an augmented transport stream.

[0035] The communications channel portion 12 includes not only thetransmission medium such as cable, terrestrial broadcast infrastructure,microwave link, or satellite link, but also any intermediate storagewhich holds the video data until received by the reception portion 14.Such intermediate broadcast storage may include video disk, tape orcassette, memory or other storage devices known to one skilled in theart. The communications channel portion also includes the headendtransmitter 50, supplied by a multiple services operator.

[0036] The reception portion 14 includes a digital receiver 54, such asa digital settop box, which decodes the signals for display on thetelevision display 58. The digital receiver hardware 54 is any digitalreceiver hardware 54 known to one skilled in the art.

[0037] In operation and referring also to FIG. 2A, a designer loadsvideo data 22 from a video source 20 into the authoring tool 24. Thevideo data 22 is also sent from the video source 20 to the video encoder36 for encoding using, for example, the MPEG standard. Using theauthoring tool 24 the designer selects portions of a video image toassociate with screen annotations. For example, the designer couldselect a shirt 2 worn by an actor in the video image and assignannotation data indicating the maker of the shirt 2, its purchase priceand the name of a local distributor. Conversely, annotation data mayinclude additional textual information about the object. For example,annotation data in a documentary program could have biographicalinformation about the individual on the screen. The annotation data 5along with information about the shape of the shirt 2 and the locationof the shirt 2 in the image, which is the mask image, as describedbelow, are stored as data structures 25, 25′ in a database 28.

[0038] Once a designer has authored a given program, the authoring tooldetermines the range over which objects appear and data structures areutilized in the annotated program. This information is used by thecurrent inventive system to ensure that the data enabling viewerinteractions with an object is transmitted before the object ispresented to the viewer. This information is also used by the currentinventive system to determine when data is no longer required by aprogram and can be erased from the memory 128 discussed below.

[0039] As described above, this annotation data is also sent to the datapacket stream generator 40 for conversion into an encoded data packetstream 27. Time stamp data in the transport stream 29 from the videoencoder 36 is also an input signal into the data packet stream generator40 and is used to synchronize the mask and the annotation data with theimage data. The data packet stream generator 40 achieves thesynchronization by stepping through a program and associating the timinginformation of each frame of video with the corresponding mask. Timinginformation can be any kind of information that allows thesynchronization of video and mask information. For example, timinginformation can be timestamp information as generated by an MPEGencoder, timecode information such as is provided by the SMPTE timecodestandard for video, frame numbering information such as a uniqueidentifier for a frame or a sequential number for a frame, the globaltime of day, and the like. In the present illustration of the invention,timestamp information will be used as an exemplary embodiment.

[0040] The encoded video data from the video encoder 36 is combined withthe encoded data packet stream 27 from the data packet stream generator40 in a multiplexer 44 and the resulting augmented transport stream 46is an input to a multiplexer system 48. In this illustrative embodimentthe multiplexer system 48 is capable of receiving additional transport29′ and augmented transport 46″ streams. The transport 29′and augmentedtransport 46′ streams include digitally encoded video, audio, and datastreams generated by the system or by other methods known in the art.The output from the multiplexer system 48 is sent to the communicationschannel 12 for storage and/or broadcast. The broadcast signal is sent toand received by the digital receiver 54. The digital receiver 54 sendsthe encoded video portion of the multiplexed signal to the television 58for display. The digital receiver 54 also accepts commands from aviewer, using a handheld remote control unit, to display any annotationsthat accompany the video images. In one embodiment the digital receiver54 is also directly in communication with an alternative networkconnection 56 (FIG. 2).

[0041] In an alternative embodiment, information from the objectdatabase 28 is transferred to a second database 30 in FIG. 2 for accessthrough a network 31, such as the Internet, or directly to a database33. In this embodiment the headend 50 accesses the annotation objectdata stored on the second database 30 when so requested by the viewer.This arrangement is useful in cases such as when the viewer has recordedthe program for viewing at a later time and the recording medium cannotrecord the annotation data, or when the data cannot be transmittedin-band during the program. Thus when the recorded image is played backthrough the digital receiver 54, and the viewer requests annotationdata, the digital receiver 54 can instruct the headend 50 to acquire thedata through the network 31. In addition, the headend 50, under thecommand of the digital receiver 54, would be able to write data to adatabase 33 on the network 31 or to a headend database 52. Such datawritten by the headend 50 may be marketing data indicating which objectshave been viewed or it could be order information required from theviewer to order the displayed item over the network. A third embodimentcombines attributes of the preceding embodiments in that some of theinformation is included in the original broadcast and some is retrievedin response to requests by the viewer.

[0042] In more detail with respect to the encoded data packet stream 27,and referring to FIGS. 2B, 2C, and 2D, the data packet stream generator40 is designed to generate a constant data rate stream despitevariations in the size of the mask data and annotation datacorresponding to the video frames. The data packet stream generator 40achieves this through a three step process. First the data packet streamgenerator 40 determines an acceptable range of packet rates which can beinputted into the multiplexer 44. Next, the data packet stream generator40 determines the number of packets filled by the largest mask in theprogram being encoded. This defines the number of packets in each maskpacket set 39. That is, the number of packets that are allocated for thetransport of each mask. In the example shown in FIGS. 2B, 2C, and 2Dthere are eight packets in each mask packet set 39. Using this number,the data packet stream generator 40 generates an initial version of theencoded data packet stream 27′, allocating a fixed number of packets foreach mask. If the number of packets required to hold a particular maskis less than the fixed number, then the data packet stream generator 40buffers the initial encoded data packet stream 27′ with null packets.The number of null packets depends on the number of packets remainingafter the mask data has been written. In FIG. 2C the mask data 42 forframe 1000 fills four packets thereby leaving four packets to be filledby null packets. Similarly the data 42′, 42″, 42′″, for mask 999, mask998, and mask 997 require three, five and two packets respectively. Thisleaves five, three, and six packets respectively to be filled by nullpackets.

[0043] Lastly, the data packet stream generator 40 generates the finalencoded data packet stream 27″ by adding the object data. The datapacket stream generator 40 does this by determining, from informationprovided by the authoring tool 24, the first occurrence that a givenobject has in a program. Data corresponding to that object is theninserted into the initial encoded data packet stream 27′ starting atsome point before the first occurrence of that object. The data packetstream generator 40 steps backwards through the initial encoded datapacket stream 27′ replacing null packets with object data as necessary.For example, in FIG. 2D object 98 is determined to appear in frame 1001.This means that all of the data associated with object 98 must arrivebefore frame 1001. The data 43 for object 98 fills five packets, O98A,O98B, O98C, O98D, and O98E, and has been added to the sets of packetsallocated to mask data 1000 and mask data 999. The data 43′ for object97 fills two packets, O97A and O97B, and has been added to the set ofpackets allocated to mask data 998.

[0044] To facilitate the process of extracting data from the transportstream in one embodiment, the multiplexer 44 associates the mask dataand the object data with different packet identifiers (PIDs) as are usedto identify elementary streams in the MPEG2 standard. In this way thedigital receiver 54 can route mask and object data to differentcomputing threads based solely on their PIDs, thereby eliminating theneed to perform an initial analysis of the contents of the packets. Inreassembling the masks and object data, the digital receiver 54 is ableto extract the appropriate number of packets from the stream becausethis information is provided by the data packet stream generator 40 aspart of the encoding process. For example referring to FIG. 2D, the datapacket stream generator 40 would specify that the mask 1000 filled fourpackets 42 and that the data 43 for object 98 filled five packets. Thisdata is included in a header 38, 38′, 38″, 38′″ portion of the packetwhich occupies the first sixteen bytes of the first packet of each maskpacket set.

[0045] As shown in an enlarged view of the mask header 38 in FIG. 2D,the header packet includes information relating to the number of packetscarrying mask information, encoding information, timestamp information,visibility word information, and the unique identifier (UID) of theobject mapping table associated with the particular mask. UIDs andobject mapping tables are discussed below in more detail with respect toFIG. 5. Similarly, the first packet for each object begins with asixteen byte header 45, 45′ that contains information that enables thedigital receiver 54 to extract, store and manipulate the data in theobject packets 43, 43′. Also, as shown in an enlarged view of the objectdata header 45 in FIG. 2D, the object data header information includesthe number of packets carrying data for the particular object, theobject's data type, the object's UID, and timestamp related informationsuch as the last instance that the object data is used in the program.The type of data structures employed by the system and the system's useof timestamps is discussed below in more detail with respect to FIGS. 5,6, and 7.

[0046] Referring to FIG. 3, the multiplexer system 8′ is an enhancedversion of the multiplexer system shown in FIG. 2A. The multiplexersystem 48′ is capable of taking multiple transport 29′ and augmentedtransport streams 46, 46″ as inputs to produce a single signal that ispassed to the broadcast medium. The illustrative multiplexer system 44′includes three transport stream multiplexers 60, 60′, 60″, threemodulators 68, 68′, 68″, three upconverters 72, 72′, 72″ and a mixer 78.The multiplexer system 44′ includes three duplicate subsystems forconverting multiple sets of transport streams into inputs to a mixer 78.Each subsystem includes a multiplexer 60, 60′, 60″ for combining a setof transport streams (TS1 to TSN), (TS1′ to TSN′), (TS1″ to TSN″) into asingle transport stream (TS, TS′, TS″) to be used as the input signal toa digital modulator (such as a Quadrature Amplitude Modulator (QAM) inthe case of a North American digital cable system or an 8VSB Modulatorin the case of terrestrial broadcast) 68, 68′, 68″. In one embodimenteach of the transport streams, for example TS1 to TSN, represent atelevision program. The output signal of the modulator 68, 68′, 68″ isan intermediate frequency input signal to an upconverter 72, 72′, 72″which converts the output signal of the modulator 68, 68′, 68″ to theproper channel frequency for broadcast. These converted channelfrequencies are the input frequencies to a frequency mixer 78 whichplaces the combined signals onto the broadcast medium.

[0047] Referring to FIG. 4, the digital receiver 54 includes a tuner 100for selecting the broadcast channel of interest from the input broadcaststream and producing an intermediate frequency (IF) signal whichcontains the video and annotation data for the channel. The IF signal isan input signal to a demodulator 104 which demodulates the IF signal andextracts the information into a transport stream (TS). The transportstream is the input signal to a video decoder 108, such as an MPEGdecoder. The video decoder 108 buffers the video frames received in aframe buffer 112. The decoded video 114 and audio 116 output signalsfrom the decoder 108 are input signals to the television display 58.

[0048] The annotation data is separated by the video decoder 108 and istransmitted to a CPU 124 for processing. The data is stored in memory128. The memory also stores a computer program for processing annotationdata and instructions from a viewer. When the digital receiver 54receives instructions from the viewer to display the annotated material,the annotation data is rendered as computer graphic images overlayingsome or all of the frame buffer 112. The decoder 108 then transmits thecorresponding video signal 114 to the television display 58.

[0049] For broadcasts carried by media which can carry signalsbi-directionally, such as cable or optical fiber, a connection can bemade from the digital receiver 54 to the headend 50 of the broadcastsystem. In an alternative embodiment for broadcasts carried byunidirectional media, such as conventional television broadcasting ortelevision satellite transmissions, a connection can be made from thedigital receiver 54 to the alternative network connection 56 thatcommunicates with a broadcaster or with another entity, without usingthe broadcast medium. Communication channels for communication with abroadcaster or another entity that are not part of the broadcast mediumcan be telephone, an internet or similar computer connection, and thelike. It should be understood that such non-broadcast communicationchannels can be used even if bi-directional broadcast media areavailable. Such communication connections, that carry messages sent fromthe viewer's location to the broadcaster or to another entity, such asan advertiser, are collectively referred to as backchannels.

[0050] Backchannel communications can be used for a variety of purposes,including gathering information that may be valuable to the broadcasteror to the advertiser, as well as allowing the viewer to interact withthe broadcaster, the advertiser or others.

[0051] In one embodiment the digital receiver 54 generates reports thatrelate to the viewer's interaction with the annotation information viathe remote control device. The reports transmitted to the broadcastervia the backchannel can include reports relating to operation of theremote, such as error reports that include information relating to useof the remote that is inappropriate with regard to the choices availableto the viewer, such as an attempt to perform an “illegal” or undefinedaction, or an activity report that includes actions taken by the viewerthat are tagged to show the timestamp of the material that was thenbeing displayed on the television display. The information that can berecognized and transmitted includes a report of a viewers' actions whenadvertiser-supplied material is available, such as actions by the viewerto access such material, as well as actions by the viewer terminatingsuch accession of the material, for example, recognizing the point atwhich a viewer cancels an accession attempt. In some embodiments, thebackchannel can be a store-and-forward channel.

[0052] The information that can be recognized and transmitted furtherincludes information relating to a transaction that a viewer wishes toengage in, for example, the placing of an order for an item advertisedon a broadcast (e.g., a shirt) including the quantity of units, thesize, the color, the viewer's credit information and/or PersonalIdentification Number (PIN), and shipping information. The informationthat can be recognized and transmitted additionally includes informationrelating to a request for a service, for example a request to be shown apay-per-view broadcast, including identification of the service, itstime and place of delivery, payment information, and the like. Theinformation that can be recognized and transmitted moreover includesinformation relating to non-commercial information, such as politicalinformation, public broadcasting information such as is provided byNational Public Radio, and requests to access data repositories, such asthe United States Patent and Trademark Office patent and trademarkdatabases, and the like.

[0053] The backchannel can also be used for interactive communications,as where a potential purchaser selects an item that is out of stock, anda series of communications ensues regarding the possibility of making analternative selection, or whether and for how long the viewer is willingto wait for the item to be restocked. Other illustrative examples ofinteractive communication are the display of a then current price,availability of a particular good or service (such as the location ofseating available in a stadium at a specific sporting event, forexample, the third game of the 2000 World Series), and confirmation of apurchase.

[0054] When a viewer begins to interact with the annotation system, thereceiver 54 can set a flag that preserves the data required to carry outthe interaction with the viewer for so long as the viewer continues theinteraction, irrespective of the programmatic material that may bedisplayed on the video display, and irrespective of a time that the datawould be discarded in the absence of the interaction by the viewer. Inone embodiment, the receiver 54 sets an “in use bit” for each datum ordata structure that appears in a data structure that is providinginformation to the viewer. A set “in use bit” prevents the receiver 54from discarding the datum or data structure. When the viewer terminatesthe interaction, the “in use bit” is reset to zero and the datum or datastructure can be discarded when its period of valid use expires. Alsopresent in the data structures of the system but not shown in FIG. 5 isa expiration timestamp for each data structure by which the systemdiscards that data structure once the time of the program has passedbeyond the expiration timestamp. This discarding process is controlledby a garbage collector 532.

[0055] In the course of interacting with the annotation system, a viewercan create and modify a catalog. The catalog can include items that theviewer can decide to purchase as well as descriptions of informationthat the viewer wishes to obtain. The viewer can make selections forinclusion in the catalog from one or more broadcasts. The viewer canmodify the contents of the catalog, and can initiate a commercialtransaction immediately upon adding an item to the catalog, or at alater time.

[0056] The catalog can include entry information about a program thatthe viewer was watching, and the number of items that were added to thecatalog. At a highest level, the viewer can interact with the system byusing a device such as a remote control to identify the item ofinterest, the ordering particulars of interest, such as quantity, price,model, size, color and the like, and the status of an order, such asimmediately placing the order or merely adding the item selected to alist of items of interest in the catalog.

[0057] At a further level of detail, the viewer can select the entry forthe program, and can review the individual entries in the catalog list,including the status of the entry, such as “saved” or “ordered.” Theentry “saved” means that the item was entered on the list but was notordered (i.e., the data pertaining to the item have been locked), while“ordered,” as the name indicates, implies that an actual order for theitem on the list was placed via the backchannel. The viewer caninterrogate the list at a still lower level of detail, to see theparticulars of an item (e.g., make, model, description, price, quantityordered, color, and so forth). If the item is not a commercial product,but rather information of interest to the viewer, for example,biographical information about an actor who appears in a scene, aninquiry at the lowest level will display the information. In oneembodiment, navigation through the catalog is performed by using theremote control.

[0058] The viewer can set up an account for use in conductingtransactions such as described above. In one embodiment, the viewer canenter information such as his name, a delivery address, and financialinformation such as a credit card or debit card number. This permits aviewer to place an order from any receiver that operates according tothe system, such as a receiver in the home of a friend or in a hotelroom. In another embodiment, the viewer can use an identifier such as asubscription account number and a password, for example the subscriptionaccount number associated with the provision of the service by thebroadcaster. In such a situation, the broadcaster already has the homeaddress and other delivery information for the viewer, as well as anopen financial account with the viewer. In such an instance, the viewersimply places an order and confirms his or her desires by use of thepassword. In still another embodiment, the viewer can set up apersonalized catalog. As an example of such a situation, members of afamily can be given a personal catalog and can order goods and servicesup to spending limits and according to rules that are pre-arranged withthe financially responsible individual in the family.

[0059] Depending on the location of the viewer and of the broadcastsystem, the format of the information conveyed over the backchannel canbe one of QPSK modulation (as is used in the United States), DVBmodulation (as is used in Europe), or other formats. Depending on theneed for security in the transmission, the messages transmitted over thebackchannel can be encrypted in whole or in part, using any encryptionmethod. The information communicated over the backchannel can includeinformation relating to authentication of the sender (for example, aunique identifier or a digital signature), integrity of thecommunication (e.g., an error correction method or system such as CRC),information relating to non-repudiation of a transaction, systems andmethods relating to prevention of denial of service, and other similarinformation relating to the privacy, authenticity, and legally bindingnature of the communication.

[0060] Depending on the kind of information that is being communicated,the information can be directed to the broadcaster, for example,information relating to viewer responses to broadcast material andrequests for pay-per-view material; information can be directed to anadvertiser, for example, an order for a shirt; and information can bedirected to third parties, for example, a request to access a databasecontrolled by a third party. FIG. 5 shows data structures that are usedin the invention for storing annotated data information. The datastructures store information about the location and/or shape of objectsidentified in video frames and information that enable viewerinteractions with identified objects.

[0061] In particular, FIG. 5 shows a frame of video 200 that includes animage of a shirt 205 as a first object, an image of a hat 206 as asecond object, and an image of a pair of shorts 207 as a third object.To represent the shape and/or location of these objects, the authoringtool 24 generates a mask 210 which is a two-dimensional pixel arraywhere each pixel has an associated integer value independent of thepixels' color or intensity value. The mask represents the locationinformation in various ways including by outlining or highlighting theobject (or region of the display), by changing or enhancing a visualeffect with which the object (or region) is displayed, by placing agraphics in a fixed relation to the object or by placing a number in afixed relation to the object. In this illustrative embodiment, thesystem generates a single mask 210 for each frame or video image. Acollection of video images sharing common elements and a common cameraperspective is defined as a shot. In the illustrative mask 210, thereare four identified regions: a background region 212 identified by theinteger 0, a shirt region 213 identified by the integer 1, a hat region214 identified by the integer 2, and a shorts region 215 identified bythe integer 3. Those skilled in the art will recognize that alternativeforms of representing objects could equally well be used, such asmathematical descriptions of an outline of the image. The mask 210 hasassociated with it a unique identifier (UID) 216, a timestamp 218, and avisibility word 219. The UID 216 refers to an object mapping table 217associated with the particular mask. The timestamp 218 comes from thevideo encoder 36 and is used by the system to synchronize the masks withthe video frames. This synchronization process is described in moredetail below with respect to FIG. 6. The visibility word 219 is used bythe system to identify those objects in a particular shot that arevisible in a particular video frame. Although not shown in FIG. 5, allthe other data structures of the system also include an in-use bit asdescribed above.

[0062] The illustrative set of data structures shown in FIG. 5 thatenable viewer interactions with identified objects include: objectmapping table 217; object properties tables 220, 220′; primary dialogtable 230; dialog tables 250, 250′, 250″; selectors 290, 290′, 290″,action identifiers 257, 257′, 257″; style sheet 240; and strings 235,235′, 235″, 235′″, 256, 256′, 256″, 259, 259′, 259″, 259′″, 259″″, 292,292′, 292″, 292′″.

[0063] The object mapping table 217 includes a region number for each ofthe identified regions 212, 213, 214, 215 in the mask 210 and acorresponding UID 216 for each region of interest. For example, in theobject mapping table 217, the shirt region 213 is stored as the integervalue “one” and has associated the UID 01234. The UID 01234 points tothe object properties table 220. Also in object mapping table 217, thehat region 214 is stored as the integer value two and has associated theUID 10324. The UID 10324 points to the object properties table 220′. Theobject mapping table begins with the integer one because the defaultvalue for the background is zero.

[0064] In general, object properties tables store references to theinformation about a particular object that is used by the system tofacilitate viewer interactions with that object. For example, the objectproperties table 220 includes a title field 221 with the UID 5678, aprice field 222 with the UID 910112, and a primary dialog field 223 withthe UID 13141516. The second object properties table 220′ includes atitle field 221′ with the UID 232323, a primary dialog field 223′ withthe same UID as the primary dialog field 223, and a price field 222′with the UID 910113. The UIDs of the title field 221 and the price field222 of object properties table 220 point respectively to strings 235,235′ that contain information about the name of the shirt, “crew poloshirt,” and its price, “$14.95.” The title field 221′ of objectproperties table 220′ points the string 235″ that contains informationabout name of the hat, “Sport Cap.” The price field of object propertiestable 220′ points to the string 235′″. Those skilled in the art willreadily recognize that for a given section of authored video numerousobject properties tables will exist corresponding to the objectsidentified by the authoring tool 24.

[0065] The UID of the primary dialog field 223 of object propertiestable 220 points to a primary dialog table 230. The dialog table 230 isalso referenced by the UID of the primary dialog field 223′ of thesecond object properties table 220′. Those skilled in the art willreadily recognize that the second object properties table 220′corresponds to another object identified within the program containingthe video frame 200. In general, dialog tables 230 structure the textand graphics boxes that are used by the system in interacting with theviewer. Dialog tables 230 act as the data model in amodel-view-controller programming paradigm. The view seen by a viewer isdescribed by a stylesheet table 240 with the UID 13579, and thecontroller component is supplied by software on the digital receiver 54.The illustrative primary dialog table 230 is used to initiateinteraction with a viewer. Additional examples of dialog tables includeboxes for indicating to the colors 250 or sizes 250′ that available fora particular item, the number of items he or she would like to purchase,for confirming a purchase 250″, and for thanking a viewer for his or herpurchase. Those skilled in the art will be aware that this list is notexhaustive and that the range of possible dialog tables is extensive.

[0066] The look and feel of a particular dialog table displayed on theviewer's screen is controlled by a stylesheet. The stylesheet controlsthe view parameters in the model-view-controller programming paradigm, asoftware development paradigm well-known to those skilled in the art.The stylesheet field 232 of dialog table 230 contains a UID 13579 thatpoints to the stylesheet table 240. The stylesheet table 240 includes afont field 241, a shape field 242, and graphics field 243. In operation,each of these fields have a UID that points to the appropriate datastructure resource. The font field 241 points to a font object, theshape field 242 points to an integer, and the graphics field 243 pointsto an image object, discussed below. By having different stylesheets,the present system is easily able to tailor the presentation ofinformation to a particular program. For example, a retailer wishing toadvertise a shirt on two programs targeted to different demographicaudiences would only need to enter the product information once. Theviewer interactions supported by these programs would reference the samedata except that different stylesheets would be used.

[0067] The name-UID pair organization of many of the data structures ofthe current embodiment provides compatibility advantages to the system.In particular, by using name-UID pairs rather than fixed fields, thedata types and protocols can be extended without affecting older digitalreceiver software and allows multiple uses of the same annotatedtelevision program.

[0068] The flexibility of the current inventive system is enhanced bythe system's requirement that the UIDs be globally unique. In theillustrative embodiment, the UIDs are defined as numbers where the firstset of bits represents a particular database license and the second setof bits represents a particular data structure element. Those skilled inthe art will recognize that this is a particular embodiment and thatmultiple ways exist to ensure that the UIDs are globally unique.

[0069] The global uniqueness of the UIDs has the advantage that, forexample, two broadcast networks broadcasting on the same cable systemcan be certain that the items identified in their programs can bedistinguished. It also means that the headend receiver 50 is able toretrieve data from databases 30, 33 over the network 31 for a particularobject because that object has an identifier that is unique across allcomponents of the system. While the global nature of the UIDs means thatthe system can ensure that different objects are distinguishable, italso means that users of the current inventive system can choose not todistinguish items when such operation is more efficient. For example, aseller selling the same shirt on multiple programs only needs to enterthe relevant object data once, and, further, the seller can use the UIDand referenced data with its supplier thereby eliminating additionaldata entry overhead.

[0070] In the present embodiment of the current system, there are fourdefined classes of UIDs: null UIDs; resource UIDs; non-resource UIDs;and extended UIDs. The null UID is a particular value used by the systemto indicate that the UID does not point to any resource. Resource UIDscan identify nine distinct types of resources: object mapping tables;object property tables; dialog tables; selectors; stylesheets; images;fonts; strings; and vectors. Selector data structures and vectorresources are discussed below. Image resources reference graphics usedby the system. The non-resource UIDs include four kinds of values: colorvalues; action identifiers; integer values; and symbols. The actionidentifiers include “save/bookmark,” “cancel,” “next item,” “previousitem,” “submit order,” and “exit,” among other actions that are taken bythe viewer. Symbols can represent names in a name-UID pair; the systemlooks up the name in the stack and substitutes the associated UID.Non-resource UIDs contain a literal value. Extended UIDs provide amechanism by which the system is able increase the size of a UID. Anextended UID indicates to the system that the current UID is the prefixof a longer UID.

[0071] When the system requires an input from the viewer it employs aselector 290 data structure. The selector 290 data structure is a tableof pairs of UIDs where a first column includes UIDs of items to bedisplayed to the viewer and a second column includes UIDs of actionsassociated with each item. When software in the digital receiver 54encounters a selector 290 data structure it renders on the viewer'sscreen all of the items in the first column, generally choices to bemade by the viewer. These items could be strings, images, or anycombination of non-resource UIDs. Once on the screen, the viewer is ableto scroll up and down through the items. If the viewer chooses one ofthe items, the software in the digital receiver 54 performs the actionassociated with that item. These actions include rendering a dialog box,performing a non-resource action identifier, or rendering anotherselector 290′data structure. Selectors are referenced by menu1 fields253, 253′, 253″, 253′″.

[0072] In operation, when a viewer selects an object and navigatesthrough a series of data structures, the system places each successivedata structure used to display information to a viewer on a stack in thememory 128. For example consider the following viewer interactionsupported by the data structures shown in FIG. 5. First a viewer selectsthe hat 214 causing the system to locate the object properties table220′ via the object mapping table 217 and to place the object propertiestable 220′ on the stack. It is implicit in the following discussion thateach data structure referenced by the viewer is placed on the stack.

[0073] Next the system displays a primary dialog table that includes thetitle 235″and price 235″ of the hat and where the style of theinformation presented to the viewer is controlled by the stylesheet 240.In addition the initial display to the viewer includes a series ofchoices that are rendered based on the information contained in theselector 290. Based on the selector 290, the system presents the viewerwith the choices represented by the strings “Exit” 256, “Buy” 256′, and“Save” 256″ each of which is respectively referenced by the UIDs 9999,8888, and 7777. The action identifiers Exit 257′ and Save 257 arereferenced to the system by the UIDs 1012 and 1020 respectively.

[0074] When the viewer selects the “Buy” string 256′, the system usesthe dialog table 250, UID 1011, to display the color options to theviewer. In particular, the selector 290′ directs the system to displayto the viewer the strings “Red” 292, “Blue” 292′, “Green” 292″, and“Yellow” 292′″, UIDs1111, 2222, 3333, 4444 respectively. The title forthe dialog table 250 is located by the system through the variableSymbol1 266. When the object properties table 220′ was placed on thestack, the Symbol1 266 was associated with the UID 2001. Therefore, whenthe system encounters the Symbol1 266′ it traces up through the stackuntil it locates the Symbol1 266 which in turn directs the system todisplay the string “Pick Color” 259 via the UID 2001.

[0075] When the viewer selects the “Blue” 2222 string 292′, the systemexecutes the action identifier associated with the UID 5555 and displaysa dialog table labeled by the string “Pick Size” 259 located throughSymbol2, UID 2002. Based on the selector 290″ located by the UID 2004,the system renders the string “Large” 259″, UID 1122, as the only sizeavailable. If the viewer had selected another color, he would have beendirected to the same dialog table, UID 5555, as the hat is onlyavailable in large. After the viewer selects the string “Large” 259″,the systems presents the viewer with the dialog table 250″, UID 6666, toconfirm the purchase. The dialog table 250″use the selector 290″, UID2003, to present to the viewer the strings “Yes” and “No”, UIDs 1113 and1114 respectively. After the viewer selects the “Yes” string 259″, thesystem transmits the transaction as directed by the action identifiersubmit order 257″, UID 1013. Had the viewer chosen the “No” strong 259′″in response to the confirmation request, the system would have exitedthe particular viewer interaction by executing the action identifierexit 257′. As part of the exit operation, the system would have dumpedfrom the stack the object properties table 220′ and all of thesubsequent data structures placed on the stack based on this particularinteraction with the system by the viewer. Similarly, after theexecution of the purchase request by the system, it would have dumpedthe data structures from the stack.

[0076] If an action requires more then one step, the system employs avector resource which is an ordered set of UIDs. For example, if aviewer wishes to save a reference to an item that he or she has located,the system has to perform two operations: first it must perform theactual save operation indicated by the non-resource save UID and secondit must present the viewer with a dialog box indicating that the itemhas been saved. Therefore the vector UID that is capable of saving areference would include the non-resource save UID and a dialog table UIDthat points to a dialog table referencing the appropriate text.

[0077] A particular advantage of the current inventive system is thatthe data structures are designed to be operationally efficient andflexible. For example, the distributed nature of the data structuresmeans that only a minimum amount of data needs to be transmitted.Multiple data structure elements, for example the object propertiestables 220, 220′, can point to the same data structure element, forexample the dialog table 230, and this data structure element only needsto be transmitted once. The stack operation described functions inconcert with the distributed nature of the data structure in that, forexample, the hat 214 does not have its own selector 290 but the selector290 can still be particularized to the hat 214 when displayed. Thedistributed nature of the data structures also has the advantage thatindividual pieces of data can be independently modified withoutdisturbing the information stored in the other data structures.

[0078] Another aspect of the current inventive system that providesflexibility is an additional use of symbols as a variable datatype. Inaddition to having a value supplied by a reference on the stack, asymbol can reference a resource that can be supplied at some time afterthe initial authoring process. For example, a symbol can direct the DTVBroadcast Infrastructure 12 to supply a price at broadcast time. Thisallows, for example, a seller to price an object differently dependingon the particular transmitting cable system.

[0079] A further aspect of the flexibility provided by the distributeddata structure of the current invention is that it supports multipleviewer interaction paradigms. For example, the extensive variation indialog tables and the ordering of their linkage means that the structureof the viewer's interaction is malleable and easily controlled by theauthor.

[0080] Another example of the variation in a viewer's experiencesupported by the system is its ability to switch between multiple videostreams. This feature exploits the structure of a MPEG2 transport streamwhich is made up of multiple program streams, where each program streamcan consist of video, audio and data information. In a MPEG2 transportstream, a single transmission at a particular frequency can yieldmultiple digital television programs in parallel. As those skilled inthe art would be aware, this is achieved by associating a programmapping table, referred to as a PMT, with each program in the stream.The PMT identifies the packet identifiers (PIDs) of the packets in thestream that correspond to the particular program. These packets includethe video, audio, and data packets for each program.

[0081] Referring to FIG. 5A, there is shown an object properties table220″ containing a link type field 270 having a corresponding link typeentry in the UID field and a stream_num field 227 with a correspondingPID 228. To enable video stream switching, the authoring tool 24 selectsthe PID 228 corresponding to the PID of a PMT 229 of a particularprogram stream. When the object corresponding to the object propertiestable 220″ is selected, the digital receiver 54 uses the video linkentry 271 of the link type field 270 to determine that the object is avideo link object. The digital receiver 54 then replaces the PID of thethen current PMT with the PID 228 of the PMT 229. The digital receiver54 subsequently uses the PMT 229 to extract data corresponding to thenew program. In particular the program referred to by the PMT 229includes a video stream 260 identified by PID17, two audio streams 261,262 identified by PID18 and PID19, and a private data stream 263identified by PID20. In this way the viewer is able to switch betweendifferent program streams by selecting the objects associated with thosestreams.

[0082]FIG. 6 is a flow diagram 500 of the data flow and flow control ofan embodiment of the system shown in FIG. 2. FIG. 6 shows sequences ofsteps that occur when a viewer interacts with the hardware and softwareof the system. In FIG. 6, a stream 502 of data for masks is decoded at amask decoder 504. The decoded mask information 506 is placed into abuffer or queue as masks 508, 508′, 508″. In parallel with the maskinformation 506, a stream 510 of events, which may be thought of asinterrupts, are placed in an event queue as events 512, 512′, 512″,where an event 512 corresponds to a mask 508 in a one-to-onecorrespondence. A thread called mask 514 operates on the masks 508. Themask 514 thread locates the mask header, assembles one or more buffers,and handles the mask information in the queue to generate mask overlays.

[0083] In order to display mask information, a thread called decompress528 decodes and expands a mask, maintained for example in (320 by 240)pixel resolution, to appropriate size for display on a video screen 530,for example using (640 by 480) pixel resolution. The decompress thread528 synchronizes the display of mask overlays to the video by examininga timestamp that is encoded in the mask information and comparing it tothe timestamp of the current video frame. If the mask overlay frame isahead of the video frame, the decompress thread sleeps for a calculatedamount of time representing the difference between the video and masktimestamps. This mechanism keeps the masks in exact synchronization withthe video so that masks appear to overlay video objects.

[0084] A second stream 516 of data for annotations is provided for asecond software thread called objects 518. The objects data stream 516is analyzed and decoded by the objects 518 thread, which decodes eachobject and incorporates it into the object hierarchy. The output of theobjects 518 thread is a stream of objects 519 that have varyingcharacteristics, such as shape, size, and the like.

[0085] A thread called model 520 combines the masks 508 and the objects519 to form a model of the system. The mask information includes aunique ID for every object that is represented in the mask overlay. Theunique IDs correspond to objects that are stored in the model. The model520 thread uses these unique IDs to synchronize, or match, thecorresponding information.

[0086] The model 520 thread includes such housekeeping structures as astack, a hash table, and a queue, all of which are well known to thoseof ordinary skill in the software arts. For example, the stack can beused to retain in memory a temporary indication of a state that can bereinstituted or a memory location that can be recalled. A hash table canbe used to store data of a particular type, or pointers to the data. Aqueue can be used to store a sequence of bits, bytes, data or the like.The model 520 interacts with a thread called view 526 that controls theinformation that is displayed on a screen 530 such as a televisionscreen. The view 526 thread uses the information contained in the model520 thread, which keeps track of the information needed to display aparticular image, with or without interactive content. The view 526thread also interacts with the mask 514 thread, to insure that theproper information is made available to the display screen 530 at thecorrect time.

[0087] A thread called soft 522 controls the functions of a statemachine called state 524. The details of state 524 are discussed morefully with regard to FIG. 7. State 524 interacts with the thread view526.

[0088] A garbage collector 532 is provided to collect and dispose ofdata and other information that becomes outdated, for example data thathas a timestamp for latest use that corresponds to a time that hasalready passed. The garbage collector 532 can periodically sweep thememory of the system to remove such unnecessary data and information andto recover memory space for storing new data and information. Suchgarbage collector software is known in the software arts.

[0089]FIG. 7 depicts the interactions between and among states withinthe state machine state 524. The state state machine 524 includes areset 602 which, upon being activated, brings the state state machine524 to a refreshed start up condition, setting all adjustable valuessuch as memory contents, stack pointers, and the like to default values,which can be stored for use as necessary in ROM, SDRAM, magneticstorage, CD-ROM, or in a protected region of memory. Once the system hasbeen reset, the state of the state state machine 524 transitions to acondition called interactive content icon 604, as indicated by arrow620. See FIG. 7.

[0090] Interactive content icon 604 is a state in which a visual image,similar to a logo, appears in a defined region of the television display58. The visual image is referred to as an “icon,” hence the nameinteractive content icon for a visual image that is active. The icon iscapable of changing appearance or changing or enhancing a visual effectwith which the icon is displayed, for example by changing color,changing transparency, changing luminosity, flashing or blinking, orappearing to move, or the like, when there is interactive informationavailable.

[0091] The viewer of the system can respond to an indication that thereis information by pressing a key on a hand-held device. For example, inone embodiment, pressing a right-pointing arrow or a left-pointing arrow(analogous to the arrows on a computer keyboard, or the volume buttonson a hand-held TV remote device) causes the state of the state statemachine 524 to change from interactive content icon 604 to maskhighlight (MH) 606. The state transition from interactive content icon604 to MH 606 is indicated by the arrow 622.

[0092] In the state MH 606, if one or more regions of an imagecorrespond to material that is available for presentation to the viewer,one such region is highlighted, for example by having a circumscribingline that outlines the region that appears on the video display (seeFIG. 1D), or by having the region or a portion thereof changeappearance. In one embodiment, if a shirt worn by a man is the objectthat is highlighted, a visually distinct outline of the shirt appears inresponse to the key press, or the shirt changes appearance in responseto the key press. Repeating the key press, or pressing the other arrowkey, causes another object, such as a wine bottle standing on a table,to be highlighted in a similar manner. In general, the objects capableof being highlighted are successively highlighted by successive keypresses.

[0093] If the viewer takes no action for a predetermined period of time,for example ten seconds, the state of the state state machine 524reverts to the interactive content icon 604 state, as denoted by thearrow 624. Alternatively, if the viewer activates a button other than asideways-pointing arrow, such as the “select” button which often appearsin the center of navigational arrows on remote controls, the stateproceeds from the state MH 606 to a state called info box 608. Info box608 is a condition wherein information appears in a pop-up box (i.e., aninformation box). The state transition from MH 606 to info box 608 isindicated by the arrow 626. The information that appears is specified byan advertiser or promoter of the information, and can, for example,include the brand name, model, price, local vendor, and specificationsof the object that is highlighted. As an example, in the case of theman's shirt, the information might include the brand of shirt, theprice, the range of sizes that are available, examples of the colorsthat are available, information about one or more vendors, informationabout special sale offers, information about telephone numbers or emailaddresses to contact to place an order, and the like.

[0094] There are many possible responses that a viewer might make, andthese responses lead, via multiple paths, back to the state interactivecontent icon 604, as indicated generally by the arrow 628. The responsescan, for example, include the viewer expressing an indication ofinterest in the information provided, as by making a purchase of theitem described, inquiring about additional information, or by decliningto make such a purchase.

[0095] While the system is in the interactive content icon 604 state,the viewer can press a burst button, which activates a state calledburst 610, causing a transition 630 from interactive content icon 604 toburst 610. In the burst 610 state, the video display automaticallyhighlights in succession all of the objects that currently haveassociated information that can be presented to a viewer. The highlightperiod of any one object is brief, of the order of 0.03 to 5 seconds, sothat the viewer can assess in a short time which objects may haveassociated information for presentation. A preferred highlight period isin the range of 0.1 to 0.5 seconds. The burst 610 state is analogous toa scan state for scanning radio receivers, in which signals that can bereceived at a suitable signal strength are successively tuned in forbrief times.

[0096] The burst 610 state automatically reverts 632 to the interactivecontent icon 604 state once the various objects that have associatedinformation have been highlighted. Once the system has returned to theinteractive content icon 604 state, the viewer is free to activate anobject of interest that has associated information, as described above.

[0097] In another embodiment, the burst 610 state can be invoked by acommand embedded within a communication. In yet another embodiment, theburst 610 state can be invoked periodically to inform a viewer of theregions that can be active, or the burst 610 state can be invoked when anew shot begins that includes regions that have set visibility bits.

[0098] The interactive content icon can be used to provide visual cluesto a viewer. In one embodiment, the interactive content icon appearsonly when there is material for display to the viewer in connection withone or more regions of an image.

[0099] In one embodiment, the interactive content icon is active whenthe burst 610 state is invoked. The interactive content icon can take ona shape that signals that the burst 610 state is beginning, for example,by displaying the interactive content icon itself in enhanced visualeffect, similar in appearance to the enhanced visual effect that eachvisible region assumes. In different embodiments, an enhanced visualeffect can be a change in color, a change in luminosity, a change in theicon itself, a blinking or flashing of a region of a display, or thelike.

[0100] In one embodiment, the interactive content icon is augmented withadditional regions, which may be shaped like pointers to points on thecompass or like keys of the digital receiver remote control. Theaugmented regions are displayed, either simultaneously or successively,with an enhanced visual effect. An illustrative example of variousembodiments are depicted schematically in FIGS. 8A through 8G. FIG. 8Adepicts an inactive interactive content icon. FIG. 8B depicts an activeinteractive content icon, that is visually enhanced. FIG. 8C depicts ainteractive content icon entering the burst state, in which fourarrowheads are added pointing to the compass positions North (N), East(E), South (S) and West (W). For example, the augmented regions can bepresented in forms that are reminiscent of the shapes of the buttons ona handheld device. In one embodiment, the North (N) and South (S)arrowheads can correspond to buttons that change channels on a videohandheld remote, and the East (E) and West (W) arrowheads can correspondto buttons that change volume on a video handheld remote, so as toremind the viewer that pushing those buttons will invoke a burst stateresponse.

[0101]FIG. 8D depicts a interactive content icon in the active burststate, in which the interactive content icon itself and the arrowheadpointing to the compass position North (N) are displayed with enhancedvisual effects. FIG. 8E depicts a interactive content icon in the activeburst state, in which the interactive content icon itself and thearrowhead pointing to the compass position East (E) are displayed withenhanced visual effects. FIG. 8F depicts a interactive content icon inthe active burst state, in which the interactive content icon itself andthe arrowhead pointing to the compass position South (S) are displayedwith enhanced visual effects. FIG. 8G depicts a interactive content iconin the active burst state, in which the interactive content icon itselfand the arrowhead pointing to the compass position West (W) aredisplayed with enhanced visual effects.

[0102] As discussed earlier, the information that appears on the videodisplay 58, including the television program and any annotationinformation that may be made available, is transmitted from a headend 50to the digital receiver 54. Video images generally contain muchinformation. In modem high definition television formats, a single videoframe may include more than 1000 lines. Each line can comprise more than1000 pixels. In some formats, a 24-bit integer is required for therepresentation of each pixel. The transmission of such large amounts ofinformation is burdensome. Compression methods that can reduce theamount of data that needs to be transmitted play a useful role intelevision communication technology. Compression of data files ingeneral is well known in the computer arts. However, new forms of filecompression are used in the invention, which are of particular use inthe field of image compression.

[0103] One traditional compression process is called “run-lengthencoding.” In this process, each pixel or group of identical pixels thatappear in succession in a video line is encoded as an ordered paircomprising a first number that indicates how many identical pixels areto be rendered and a second number that defines the appearance of eachsuch identical pixel. If there are long runs of identical pixels, such acoding process can reduce the total number of bits that must betransmitted. However, in pathological instances, for example where everypixel differs from the pixel that precedes it and the pixel that followsit, the coding scheme can actually require more bits that the number ofbits required to represent the pixel sequence itself.

[0104] In one embodiment, an improvement on run-length encoding, called“Section Run-Length Encoding,” is obtained if two or more successivelines can be categorized as having the same sequence of run lengths withthe same sequence of appearance or color. The two or more lines aretreated as a section of the video image. An example of such a section isa person viewed against a monochrome background. A transmitter encodesthe section by providing a single sequence of colors that is valid forall lines in the section, and then encodes the numbers of pixels perline that have each successive color. This method obviates the repeatedtransmission of redundant color information which requires a lengthy bitpattern per color.

[0105]FIG. 9A depicts an image 700 of a person 705 shown against amonochrome background 710, for example, a blue background. FIG. 9Aillustrates several embodiments of compression methods for video images.In FIG. 9A the person has a skin color which is apparent in the region720. The person is wearing a purple shirt 730 and green pants 740.Different colors or appearances can be encoded as numbers having smallvalues, if the encoder and the decoder use a look-up table to translatethe coded numbers to fill (e.g., 24-bit) display values. As oneembodiment, a background color may be defined, for the purposes of amask, as a null, or a transparent visual effect, permitting the originalvisual appearance of the image to be displayed without modification.

[0106] In this embodiment of “Section Run-Length Encoding,” the encoderscans each row 752, 754, 762, 764, and records the color value andlength of each run. If the number of runs and the sequence of colors ofthe first row 752 of a video frame does not match that of the succeedingrow 754, the first row 752 is encoded as being a section of length 1,and the succeeding row 754 is compared to the next succeeding row. Whentwo or more rows do contain the same sequence of colors, the section isencoded as a number of rows having the same sequence of colors, followedby a series of ordered pairs representing the colors and run lengths forthe first row of the section. As shown in FIG. 9B for an example havingthree rows, the first row includes (n) values of pairs of colors and runlengths. The remaining two rows are encoded as run lengths only, and thecolors used in the first row of the section are used by a decoder toregenerate the information for displaying the later rows of the section.In one embodiment, the section can be defined to be less than the entireextent of a video line or row.

[0107] As an example expressed with regard to FIG. 9A, the illustrativerows 752 and 754, corresponding to video scan lines that include thebackground 710, a segment of the person's skin 720, and additionalbackground 710. The illustrative rows 752 and 754 both comprise runs ofblue pixels, skin-colored pixels, and more blue pixels. Thus, the rows752 and 754, as well as other adjacent rows that intersect theskin-colored head or neck portion of the person, would be encoded asfollows: a number indicating exactly how many rows similar to the lines752, 754 are in a section defined by the blue background color-skincolor-blue background color pattern; a first row encoding comprising thevalue indicative of blue background color and an associated pixel count,the value indicative of skin color 720 and an associated pixel count,and the value indicative of blue background color and another associatedpixel count. The remaining rows in the section would be encoded as anumber representing a count of blue background color pixels, a numberrepresentative of a count of pixels to be rendered in skin color 720,and a number representing the remaining blue background color pixels.

[0108] In another embodiment, a process that reduces the informationthat needs to be encoded, called “X-Run-Length Encoding,” involvesencoding only the information within objects that have been identified.In this embodiment, the encoded pixels are only those that appear withinthe defined object, or within an outline of the object. An encoder in atransmitter represents the pixels as an ordered triple comprising avalue, a run length and an offset defining the starting position of therun with respect to a known pixel, such as the start of the line. In areceiver, a decoder recovers the encoded information by reading theordered triple and rendering the pixels according to the encodedinformation.

[0109] Referring again to FIG. 9A, each of the illustrative lines 752and 754 are represented in the X-Run-Length Encoding process as anordered triple of numbers, comprising a number indicative of the skincolor 720, a number representing how many pixels should be rendered inskin color 720, and a number indicative of the distance from one edge712 of the image 700 that the pixels being rendered in skin color 720should be positioned. An illustrative example is given in FIG. 9C.

[0110] In yet another embodiment, a process called “X-Section-Run-LengthEncoding,” that combines features of the Section Run-Length andX-Run-Length encoding processes is employed. The X-Section-Run-LengthEncoding process uses color values and run lengths as coding parameters,but ignores the encoding of background. Each entry in this encodingscheme is an ordered triple of color, run length, and offset values asin X-Run-Length Encoding.

[0111] The illustrative lines 762 and 764 are part of a section ofsuccessive lines that can be described as follows: the illustrativelines 762, 764 include, in order, segments of blue background 710, anarm of purple shirt 730, blue background 710, the body of purple shirt730, blue background 710, the other arm of purple shirt 730, and a finalsegment of blue background. Illustrative lines 762, 764 and the otheradjacent lines that have the same pattern of colors are encoded asfollows: an integer defining the number of lines in the section; thefirst line is encoded as three triples of numbers indicating a color, arun length and an offset; and the remaining lines in the section areencoded as three ordered doubles of numbers indicating a run length andan offset. The color values are decoded from the sets of triples, andare used thereafter for the remaining lines of the section. Pixels whichare not defined by the ordered doubles or triples are rendered in thebackground color. An illustrative example is shown in FIG. 9D, usingthree rows.

[0112] A still further embodiment involves a process called“Super-Run-Length Encoding.” In this embodiment, a video image isdecomposed by a CPU into a plurality of regions, which can includesections. The CPU applies the compression processes described above tothe various regions, and determines an encoding of the most efficientcompression process on a section-by section basis. The CPU then encodesthe image on a section-by-section basis, as a composite of the mostefficient processes, with the addition of a prepended integer or symbolthat indicates the process by which each section has been encoded. Anillustrative example of this Super-Run-Length Encoding is the encodingof the image 700 using a combination of run length encoding for somelines of the image 700, X Run-Length Encoding for other lines (e.g.,752, 754) of image 700, X-Section-Run-Length Encoding for still otherlines (e.g., 762, 764) of image 700, and so forth.

[0113] Other embodiments of encoding schemes may be employed. Oneembodiment that may be employed involves computing an offset of thepixels of one line from the preceding line, for example shifting asubsequent line, such as one in the vicinity of the neck of the persondepicted in FIG. 9A, by a small number of pixels, and filling anyundefined pixels at either end of the shifted line with pixelsrepresenting the background. This approach can be applied to both runlengths and row position information. This embodiment provides anadvantage that an offset of seven or fewer pixels can be represented asa signed four-bit value, with a large savings in the amount ofinformation that needs to be transmitted to define the line so encoded.Many images of objects involve line to line offsets that are relativelymodest, and such encoding can provide a significant reduction in data tobe transmitted.

[0114] Another embodiment involves encoding run values within theconfines of an outline as ordered pairs, beginning at one edge of theoutline. Other combinations of such encoding schemes will be apparent tothose skilled in the data compression arts.

[0115] In order to carry out the objectives of the invention, an abilityto perform analysis of the content of images is useful in addition torepresenting the content of images efficiently. Television imagescomprising a plurality of pixels can be analyzed to determine thepresence or absence of persons, objects and features, so thatannotations can be assigned to selected persons, objects and features.The motions of persons, objects and features can also be analyzed. Anassignment of pixels in an image or a frame to one or more persons,objects, and/or features is carried out before such analysis isperformed.

[0116] The analysis is useful in manipulating images to produce a smoothimage, or one which is pleasing to the observer, rather than an imagethat has jagged or rough edges. The analysis can also be used to definea region of the image that is circumscribed by an outline having adefined thickness in pixels. In addition, the ability to define a regionusing mathematical relationships makes possible the visual modificationof such a region by use of a visibility bit that indicates whether theregion is visible or invisible, and by use of techniques that allow therendering of all the pixels in a region in a specific color or visualeffect. An image is examined for regions that define matter that is ofinterest. For example, in FIG. 9A, a shirt region 730, a head region720, and a pants region 740 are identified.

[0117] In one embodiment, the pixels in an image or frame are classifiedas belonging to a region. The classification can be based on theobservations of a viewer, who can interact with an image presented indigital form on a digital display device, such as the monitor of acomputer. In one embodiment, the author/annotator can mark regions of animage using an input device such as a mouse or other computer pointingdevice, a touch screen, a light pen, or the like. In another embodiment,the regions can be determined by a computing device such as a digitalcomputer or a digital signal processor, in conjunction with software. Ineither instance, there can be pixels that are difficult to classify asbelonging to a region, for example when a plurality of regions abut oneanother.

[0118] In one embodiment, a pixel that is difficult to classify, orwhose classification is ambiguous, can be classified by a process thatinvolves several steps. First, the classification of the pixel iseliminated, or canceled. This declassified pixel is used as the point oforigin of a classification shape that extends to cover a plurality ofpixels (i.e., a neighborhood) in the vicinity of the declassified pixel.The pixels so covered are examined for their classification, and theambiguous pixel is assigned to the class having the largestrepresentation in the neighborhood. In one embodiment, the neighborhoodcomprises next nearest neighbors of the ambiguous pixel. In oneembodiment, a rule is applied to make an assignment in the case of tiesin representation. In one embodiment, the rule can be to assign theclass of a pixel in a particular position relative to the pixel, such asthe class of the nearest neighbor closest to the upper left hand cornerof the image belonging to a most heavily represented class.

[0119] In another embodiment, a pixel that is difficult to classify, orwhose classification is ambiguous, can be classified by a process whichfeatures a novel implementation of principles of mathematicalmorphology. Mathematical morphology represents the pixels of an image inmathematical terms, and allows the algorithmic computation of propertiesand transformations of images, for example, using a digital computer ordigital signal processor and appropriate software. The principles ofmathematical morphology can be used to create various image processingapplications. A very brief discussion of some of the principles will bepresented here. In particular, the methods known as dilation and erosionwill be described and explained. In general, dilation and erosion can beused to change the shape, the size and some features of regions Inaddition, some illustrative examples of applications of the principlesof mathematical morphology to image processing will be described.

[0120] Dilation and erosion are fundamental mathematical operations thatact on sets of pixels. As an exemplary description in terms of an imagein two-dimensional space, consider the set of points of a region R, anda two-dimensional morphological mask M. The illustrative discussion,presented in terms of binary mathematical morphology, is given withrespect to FIGS. 10A and 10B. In FIG. 10A, the morphological mask M hasa shape, for example, a five pixel array in the shape of a “plus” sign.Morphological masks of different shape can be selected depending on theeffect that one wants to obtain. The region R can be any shape; forpurposes of illustration, the region R will be taken to be the irregularshape shown in FIG. 10A.

[0121] The morphological mask M moves across the image in FIG. 10A, andthe result of the operation is recorded in an array, which can berepresented visually as a frame as shown in FIG. 10 B. For theillustrative morphological mask, the pixel located at the intersectionof the vertical and the horizontal lines of the “plus” sign is selectedas a “test” pixel, or the pixel that will be “turned on” (e.g., setto 1) or “turned off” (e.g., set to 0) according to the outcome of theoperation applied.

[0122] For binary erosion, the mathematical rule, expressed in terms ofset theory, can be that the intersection of one or more pixels of themorphological mask M with the region R defines the condition of thepixel to be stored in an array or to be plotted at the position in FIG.10B corresponding to the location of the test pixel in FIG. 10A. Thisrule means that, moving the morphological mask one pixel at a time, ifall the designated pixel or pixels of the morphological mask M intersectpixels of the region R, the test pixel is turned on and thecorresponding pixel in FIG. 10B is left in a turned on condition. Thescanning of the mask can be from left to right across each row of theimage, starting at the top row and moving to the bottom, for example.Other scan paths that cover the entire image (or at least the region ofinterest) can be used, as will be appreciated by those of ordinary skillin the mathematical morphology arts. This operation tends to smooth aregion, and depending on the size and shape of the morphological mask,can have a tendency to eliminate spiked projections along the contoursof a region. Furthermore, depending on the size and shape of themorphological mask, an image can be diminished in size.

[0123] Binary dilation can have as a mathematical rule, expressed interms of set theory, that the union of the morphological mask M with theregion R defines the condition of the pixel to be plotted at theposition in FIG. 10B corresponding to the location of the test pixel inFIG. 10A. For a given location of the morphological mask M, the pixelsof R and the pixels of M are examined, and if any pixel turned on in Mcorresponds to a pixel turned on in R, the test pixel is turned on. Thisrule is also applied by scanning the morphological mask across the imageas described above, for example, from left to right across each row ofthe image, again from the top row to the bottom. This operation can havea tendency to cause a region to expand and fill small holes. Theoperations of dilation and erosion are not commutative, which means thatin general, one obtains different results for applying erosion followedby dilation as compared to applying dilation followed by erosion.

[0124] The operations of erosion and dilation, and other operationsbased upon these fundamental operations, can be applied to sets ofpixels defined in space, as are found in a two-dimensional image, as hasjust been explained. The same operations can be applied equally well forsets of pixels in a time sequence of images, as is shown in FIGS. 11Aand 11B. In FIG. 11A, time may be viewed as a third dimension, which isorthogonal to the two dimensions that define each image or frame. FIG.11A shows three images or frames, denoted as N−1, N, and N+1, whereframe N−1 is displayed first, frame N appears next, and finally frameN+1 appears. Each frame can be thought of as having an x-axis and ay-axis In an illustrative example, each frame comprises 480 horizontalrows of 640 pixels, or columns each. It is conventional to number rowsfrom the top down, and to number columns from the left edge and proceedto the right. The upper left hand corner is row 0, column 0, or (0,0).The x-axis defines the row, with increasing x value as one movesdownward along the left side of the frame, and the y-axis defines thecolumn number per row, with increasing y value as one moves rightwardalong the top edge of the frame. The time axis, along which timeincreases, is then viewed as proceeding horizontally from left to rightin FIG. 11A.

[0125] The operations of erosion and dilation in two-dimensional spaceused a morphological mask, such as the five-pixel “plus” sign, which isoriented in the plane of the image or frame. An operation in the timedimension that uses the two-dimensional five-pixel “plus” sign as amorphological mask can be understood as in the discussion that follows,recognizing that one dimension of the “plus” sign lies along the timeaxis, and the other lies along a spatial axis. In other embodiments, onecould use a one dimensional morphological mask along only the time axis,or a three-dimensional morphological mask having dimensions in twonon-collinear spatial directions and one dimension along the time axis.

[0126] Let the “test” pixel of the two-dimensional five-pixel “plus”sign morphological mask be situated at row r, column c, or location(r,c), of frame N in FIG. 11A. The pixels in the vertical line of the“plus” sign is at column c of row r−1 (the row above row r) of frame Nand column c of row r+1 (the row below row r) of frame N. The pixel tothe “left” of the “test” pixel is at row r, column c of frame N−1 ofFIG. 11A (the flame preceding frame N), and the pixel to the “right” ofthe “test” pixel is at row r, column c of frame N+1 of FIG. 11A (theframe following frame N). An operation using this morphological maskthus has its result recorded visually at row r, column c of a framecorresponding to frame N, and the result can be recorded in an array atthe corresponding location. However, in this example, the computationdepends on three pixels situated in frame N, one pixel situated in frameN−1, and one situated in frame N+1. FIG. 11A schematically depicts theuse of the five-pixel “plus” mask on three images or frames thatrepresent successive images in time, and FIG. 11B depicts the result ofthe computation in a frame corresponding to frame N.

[0127] In this inventive system, a novel form of erosion and dilation isapplied in which all regions are eroded and dilated in one pass, ratherthan working on a single region at a time (where the region is labeled‘1’ and the non-region is ‘0’), and repeating the process multiple timesin the event that there are multiple regions to treat. In the case oferosion, if the input image contains R regions, the pixels of which arelabeled 1, 2, . . . r, respectively, then the test pixel is labeled, forexample, ‘3’, if and only if all the pixels under the set pixels in themorphological mask are labeled 3. Otherwise, the test pixel is assigned0, or “unclassified.” In the case of dilation, if the input imagecontains R regions, the pixels of which are labeled 1, 2, . . . r,respectively, then the test pixel is labeled, for example, ‘3’, if andonly if the region with the greatest number of pixels is the one withlabel 3. Otherwise, the test pixel is assigned 0, or “unclassified.”

[0128] Two dimensional floodfill is a technique well known in the artthat causes a characteristic of a two-dimensional surface to be changedto a defined characteristic. For example, two-dimensional floodfill canbe used to change the visual effect of a connected region of an image tochange in a defined way, for example changing all the pixels of theregion to red color. Three-dimensional floodfill can be used to changeall the elements of a volume to a defined characteristic. For example, avolume can be used to represent a region that appears in a series ofsequential two-dimensional images that differ in sequence number or intime of display as the third dimension.

[0129] An efficient novel algorithm has been devised to floodfill aconnected three-dimensional volume starting with an image that includesa region that is part of the volume. In overview, the method allows theselection of an element at a two-dimensional surface within the volume,and performs a two-dimensional floodfill on the region containing thatselected element. The method selects a direction along the thirddimension, determines if a successive surface contains an element withinthe volume, and if so, performs a two-dimensional floodfill of theregion containing such an element. The method repeats the process untilno further elements are found, and returns to the region firstfloodfilled and repeats the process while moving along the thirddimension in the opposite direction.

[0130] An algorithmic image processing technique has been devised usinga three-dimensional flood-fill operator in which the author selects apoint within a group of incorrectly classified points. The selectedpoint can be reclassified using a classification method as describedearlier. The entire group of pixels contiguous with the selected pointis then reclassified to the classification of the selected point. Pixelsthat neighbor the reclassified pixels in preceding and following framescan also be reclassified.

[0131] In one embodiment, the three-dimensional volume to bereclassified comprises two dimensions representing the image plane, anda third dimension representing time. In this embodiment, for every pixel(r, c) in frame N of FIG. 11A that has changed from color A to color Bdue to the two-dimensional floodfill operation in frame N, if pixel (r,c) in frame N+1 of FIG. 11A is currently assigned color A, then thetwo-dimensional floodfill is run starting at pixel (r, c) in frame N+1of FIG. 11A, thereby changing all the contiguous pixels in frame N+1assigned to color A. Again with reference to FIG. 11A, it is equallypossible to begin such a process at frame N and proceed backward in thetime dimension to frame N−1. In one embodiment, the three-dimensionalfloodfill process is terminated at a frame in which no pixel has a labelthat requires changing as a result of the flood fill operation. In oneembodiment, once three-dimensional floodfill is terminated going in onedirection in time, the process is continued by beginning at the initialframe N and proceeding in the opposite direction in time until theprocess terminates again.

[0132]FIG. 11C is a flow diagram 1150 showing an illustrative process bywhich three-dimensional floodfill is accomplished, according to oneembodiment of the invention. The process starts at the circle 1152labeled “Begin.” The entity that operates the process, such as anoperator of an authoring tool, or alternatively, a computer thatanalyzes images to locate within images one or more regionscorresponding to objects, selects a plurality of sequentialtwo-dimensional sections that circumscribe the volume to be filled inthe three-dimensional floodfill process, as indicated by step 1154. Inone embodiment, the three-dimensional volume comprises two dimensionalsections disposed orthogonally to a third dimension, eachtwo-dimensional section containing locations identified by a firstcoordinate and a second coordinate. For example, in one embodiment, thetwo-dimensional sections can be image frames, and the third dimensioncan represent time or a frame number that identifies successive frames.In one embodiment, the first and second coordinates can represent rowand column locations that define the location of a pixel within an imageframe on a display.

[0133] In step 1156, the process operator defines a plurality of regionsin at least one of the two-dimensional sections, each region comprisingat least one location. From this point forward in the process, theprocess is carried out using a machine such as a computer that canperform a series of instructions such as may be encoded in software. Thecomputer can record information corresponding to the definitions forlater use, for example in a machine-readable memory. For example, in animage, an operator can define a background and an object of interest,such as shirt 2.

[0134] In step 1158, the computer selects a first region in one of thetwo-dimensional sections, the region included within the volume to befilled with a selected symbol. In one embodiment, the symbol can be avisual effect when rendered on a display, such as a color, ahighlighting, a change in luminosity, or the like, or it can be acharacter such as an alphanumeric character or another such symbol thatcan be rendered on a display.

[0135] In step 1160, the computer that runs the display fills the firstregion with the selected symbol. There are many different well-knowngraphics routines for filling a two-dimensional region with a symbol,such as turning a defines region of a display screen to a defined color.Any such well-known two-dimensional graphics routine can be implementedto carry out the two-dimensional filling step.

[0136] In step 1162, the computer moves in a first direction along thethird dimension to the successive two-dimensional section. In oneembodiment, the process operator moves to the image immediately beforeor after the first image selected, thus defining a direction in time, orin the image sequence.

[0137] In step 1164, the computer determines whether a location in thesuccessive two-dimensional section corresponding to a filled location inthe two-dimensional section of the previous two-dimensional sectionbelongs to the volume. The process operator looks up informationrecorded in the definitions of the two dimensional regions accomplishedinstep 1156.

[0138] In step 1166, the computer makes a selection based on the outcomeof the determination performed in step 1164. If there is a positiveoutcome of the determination step 1164, the computer fills a region thatincludes the location in the successive two-dimensional section with theselected symbol, as indicated at step 1168. As indicated at step 1170,beginning with the newly-filled region in the successive two-dimensionalsection, the computer repeats the moving step 1162, the determining step1164 and the filling step 1168 (that is, the steps recited immediatelyheretofore) until the determining step results in a negative outcome.

[0139] Upon a negative outcome of any determining step 1164 heretofore,the computer returns to the first region identified in step 1158 (whichhas already been filled), and, moving along the third dimension in adirection opposite to the first direction, repeating the steps of moving(e.g., a step similar to step 1162 but going on the opposite direction),determining (e.g., a step such as step 1164) and filling (e.g., a stepsuch as step 1168) as stated above until a negative outcome results fora determining step. This sequence is indicated in summary form at step1172. At step 1174, the process ends upon a negative outcome of adetermining step.

[0140] Another application involves creating outlines of regions, forexample to allow a region to be highlighted either in its entirety, orto be highlighted by changing the visual effect associated with theoutline of the region, or some combination of the two effects. In oneembodiment, a method to construct outlines from labeled regions isimplemented as depicted in FIGS. 12A-12B, A region 1210 to be outlinedhaving an outline 1215 in input image 1218 is shown in FIG. 12A A squaremorphological mask 1220 having an odd number of pixels whose size isproportional to the desired outline thickness is passed over the region1210. At every position in the input region 1210, the pixels fallingwithin the morphological mask are checked to see if they are all thesame. If so, a ‘0’ is assigned to the test pixel in the output image1230 of FIG. 12B. If a pixel is different from any other pixel withinthe morphological mask, then the label which falls under themorphological mask's center pixel is assigned to the test pixel in theoutput image 1230. As the morphological mask 1220 passes over the region1210, a resulting outline 1215′ is generated in output image 1230. Inother embodiments, square morphological masks having even numbers ofpixels, morphological masks having shapes other than square, and squaremorphological masks having odd numbers of pixels can be used in whichone selects a particular pixel within the morphological mask as thepixel corresponding to the test pixel 1222 in the output image 1230.

[0141] It will be understood that those of ordinary skill in using theprinciples of mathematical morphology may construct the foregoingexamples of applications by use of alternative morphological masks, andalternative rules, and will recognize many other similar applicationsbased on such principles.

[0142] In a series of related images, or a shot as described previously,such as a sequence of images showing a person sitting on a bench in thepark, one or more of the selected objects may persist for a number offrames. In other situations, such as an abrupt change in the image, aswhere the scene changes to the view perceived by the person sitting onthe park bench, some or all of the regions identified in the first sceneor shot may be absent in the second scene or shot. The system and methodof the invention can determine both that the scene has changed (e.g., anew shot begins) and that one or more regions present in the first sceneare not present in the second scene.

[0143] In one embodiment, the system and method determines that thescene or shot has changed by computing a histogram of pixels that havechanged from one image to a successive image and comparing the slope ofthe successive instances (or time evolution) of the histogram to apredefined slope value. FIG. 13 shows three illustrative examples of theevolutions of histograms over successive frames (or over time). Thetopmost curve 1310 has a small variation in slope from zero andrepresents motion at moderate speed. The middle curve 1320 shows asomewhat larger variation in slope and represents sudden motion. Thelowermost curve 1330 shows a large variation in slope, and represents ashot change or scene change at frame F. If the slope of the histogramevolution plot exceeds a predetermined value, as does the lowermostcurve 1330, the system determines that a shot change has occurred.

[0144] While the invention has been particularly shown and describedwith reference to specific preferred embodiments, it should beunderstood by those skilled in the art that various changes in form anddetail may be made therein without departing from the spirit and scopeof the invention as defined by the appended claims.

What is claimed is:
 1. A hyperlinked broadcast system comprising: avideo source; a video encoder in communication with said video source,said video encoder producing a transport stream; an annotation source; adata packet stream generator in communication with said annotationsource and said video encoder, said data packet stream generatorproducing encoded data packets; and a multiplexer system incommunication with said video encoder and said data packet streamgenerator, said multiplexer generating a digital broadcast signalcomprising an augmented transport stream from said transport stream andsaid encoded data packets, wherein said video encoder provides timinginformation to said data packet stream generator and said data packetstream generator synchronizes annotation data from said annotationsource with a video signal from said video source in response to saidtiming information.
 2. The system of claim 1 wherein said timinginformation comprises one of timestamp information, timecodeinformation, frame numbering information and global time of day.
 3. Thesystem of claim 1 wherein said annotation data comprises mask data andat least one of textual data and graphics data.
 4. The system of claim 3wherein said mask data includes location information of an object in anannotated video frame.
 5. The system of claim 4 wherein said locationinformation includes a graphics location reference that represents afixed relation to a set of pixels associated with said object.
 6. Thesystem of claim 5 wherein said graphics location reference includes anupper left most pixel in said associated pixel set.
 7. The system ofclaim 5 wherein said graphics location reference includes a centroidpixel of said associated pixel set.
 8. The system of claim 3 whereinsaid mask data comprises location and shape information of an object ina video fame to be annotated.
 9. The system of claim 8 wherein saidshape information is represented by a graphical overlay of said object.10. The system of claim 8 wherein said shape information is representedby an outline of said object.
 11. The system of claim 8 wherein saidshape information is represented by a mathematical representation of aset of pixels associated with said object.
 12. A hyperlinked broadcastand reception system comprising: a video source; a video encoder incommunication with said video source, said video encoder producing atransport stream; an annotation source; a data packet stream generatorin communication with said annotation source and said video encoder,said data packet stream generator producing encoded data packets; amultiplexer system in communication with said video encoder and saiddata packet stream generator, said multiplexer generating a digitalbroadcast signal comprising an augmented transport stream from saidtransport stream and said encoded data packets; a broadcast channel incommunication with said multiplexer system; a receiver in communicationwith said broadcast channel; and a display device in communication withsaid receiver, wherein said video encoder provides timing information tosaid data packet stream generator and said data packet stream generatorsynchronizes annotation data from said annotation source with a videosignal from said video source in response to said timing information,and wherein said display device displays said annotation data associatedwith said video signal in response to a viewer request on a frame byframe basis.
 13. The system of claim 12 wherein said timing informationcomprises one of timestamp information, timecode information, framenumbering information and global time of day.
 14. The system of claim12, wherein said annotation data comprises mask data and at least one oftextual data and graphics data.
 15. The system of claim 14, wherein saidmask data and said at least one of textual data and graphics data aretransmitted over separate streams in said broadcast channel.
 16. Thesystem of claim 14 wherein said mask data comprises location informationof an object in an annotated video frame.
 17. The system of claim 16wherein said mask data further comprises shape information of an objectin an annotated video frame.
 18. The system of claim 12 wherein saidtiming information is employed to synchronize the display of annotationdata from said annotation source with said video signal from said videosource.
 19. The system of claim 18 wherein said timing informationcomprises one of timestamp information, timecode information, framenumbering information and global time of day.
 20. The system of claim 12wherein said system transmits information related to said viewer requestto a remote location over a backchannel communications channel.
 21. Thesystem of claim 20 wherein said backchannel communications channel is astore-and-forward channel.
 22. The system of claim 12, wherein saidannotation data comprises information regarding goods and services forsale.
 23. The system of claim 12, wherein said annotation data comprisesnon-commercial information.
 24. A hyperlinked reception systemcomprising: a receiver in communication with a broadcast channel; and adisplay device in communication with said receiver, wherein saidreceiver decodes a digital broadcast signal to recover a video signaland annotation data; and wherein, in response to a viewer request, saiddisplay device displays said annotation data associated with said videosignal on a frame by frame basis.
 25. The system of claim 24 whereintiming information is employed to synchronize a display of saidannotation data with said video signal.
 26. The system of claim 25wherein said timing information comprises one of timestamp information,timecode information, frame numbering information and global time ofday.
 27. The system of claim 25 wherein said receiver synchronizes saidannotation data with said video signal in response to said timinginformation.
 28. The system of claim 24 wherein said annotation datacomprises a plurality of annotations having equal timing information,and said viewer request comprises an indication as to which annotationof said plurality of annotations is to be displayed.
 29. A hyperlinkedreception system comprising: a receiver in communication with abroadcast channel; and a display device in communication with saidreceiver, wherein said display device displays annotation dataassociated with a video signal, in response to a viewer request, on aframe by frame basis, said annotation data being associated with saidvideo signal in response to timing information.
 30. The system of claim29 wherein said timing information comprises one of timestampinformation, timecode information, frame numbering information andglobal time of day.
 31. A method of generating a hyperlinked videosignal, comprising: providing a source video; encoding said source videowith a video encoder to produce a transport stream and timinginformation; providing a source of annotation data; communicating saidtransport stream and timing information from said video encoder and saidannotation data from said source of annotation data to a data packetstream generator, said data packet stream generator synchronizing saidannotation data with a video signal from said video source in responseto said timing information and producing encoded data packets.
 32. Themethod of claim 31 wherein said timing information comprises one oftimestamp information, timecode information, frame numbering informationand global time of day.
 33. The method of claim 31, wherein saidannotation data comprises information regarding goods and services forsale.
 34. The method of claim 31, wherein said annotation data comprisesnon-commercial information.
 35. The method of claim 31 wherein saidannotation data comprises mask data and at least one of textual data andgraphics data.
 36. The method of claim 35 wherein said mask dataincludes location information of an object in an annotated video frame.37. The method of claim 36 wherein said location information includes agraphics location reference that represents a fixed relation to a set ofpixels associated with said object.
 38. The method of claim 37 whereinsaid graphics location reference includes an upper left most pixel insaid associated pixel set.
 39. The method of claim 38 wherein saidgraphics location reference includes a centroid pixel of said associatedpixel set.
 40. The method of claim 35 wherein said mask data compriseslocation and shape information of an object in an annotated video frame.41. The method of claim 40 wherein said shape information is representedby a graphical overlay of said object.
 42. The method of claim 40wherein said shape information is represented by an outline of saidobject.
 43. The method of claim 40 wherein said shape information isrepresented by a mathematical representation of a set of pixelsassociated with said object.
 44. The method of claim 31, furthercomprising: communicating said transport stream and said encoded datapackets to a multiplexer system, said multiplexer system generating ahyperlinked digital broadcast signal comprising an augmented transportstream from said transport stream and said encoded data packets.
 45. Themethod of claim 44 wherein said annotation data comprises mask data andat least one of textual data and graphics data and wherein said maskdata and said at least one of textual data and graphics data representseparate streams contained in said hyperlinked digital broadcast signal.46. The method of claim 44, further comprising: broadcasting saidhyperlinked digital broadcast signal.
 47. A method of viewing ahyperlinked video signal, comprising: receiving a hyperlinked videosignal comprising an augmented transport stream; demultiplexing saidaugmented transport stream to recover a transport stream and encodeddata packets; displaying said transport stream framewise as a video on adisplay; decoding said data packets to recover annotation data andtiming information; synchronizing said annotation data with said videousing said timing information; and in response to a viewer request,displaying said annotation data on said display on a frame by framebasis.
 48. The method of claim 47 wherein said timing informationcomprises one of timestamp information, timecode information, framenumbering information and global time of day.
 49. The method of claim47, further comprising transmitting information related to said viewerrequest to a remote location over a backchannel communications channel.50. The method of claim 47, further comprising: in response to a viewerprompt, accepting from a viewer information regarding a commercialtransaction, and completing said transaction.
 51. The method of claim50, wherein the information regarding a commercial transaction acceptedfrom a viewer comprises at least one of a viewer's identifier, aviewer's address, an identifier of a viewer's financial account, aviewer's password, a viewer's personal identification number, aquantity, a model, a size, a color, a descriptor of a good or service tobe vended, and a price.